Generative Facial Prior and Semantic Guidance for Iterative Face Inpainting

Image inpainting techniques have been greatly improved by relying on structure and texture priors. However, damaged original images or rough predictions cannot provide sufficient texture information and accurate structural priors, leading to a drop in image quality. Moreover, from the perspective of human visual perception, it is important to pay attention to facial symmetry and facial attribute consistency. In this paper, we present a face inpainting system with iteration structure, guided by generative facial priors contained in pretrained GANs and predicted semantic information. Specifically, generative facial priors generated by the GAN inversion techniques introduce sufficient textures and features to assist inpainting;semantic maps are able to provide facial structural information and semantic categories of different pixels for face reconstruction. In particular, we iteratively refine images multiple times, updating semantic maps at each iteration. The Weighted Prior-Guidance Modulation layer (WPGM) is devised for incorporating priors into networks through spatial modulation. We also propose facial feature self-symmetry loss to constrain the symmetry of faces in feature space. Experiments on CelebA-HQ and LaPa datasets demonstrate the superiority of our model for facial detail and attribute consistency. Meanwhile, under the background of COVID-19, it is worth trying recognition via inpainting to deal with recognition challenges brought by mask occlusion. Relevant experiments show that our inpainting model does help to recognition tasks to a certain degree, with higher accuracy.

Lower Face Inpainting Aiming at Face Recognition under Occlusions

Although both face recognition and object inpainting have become promising approaches through the use of deep learning, the COVID-19 pandemic has created a tremendous challenge to their further development. Masks, which people have become accustomed to as an effective sanitary measure to prevent infection of COVID-19, have also become an undeniable physical barrier between devices applying face recognition authentication and the faces to be recognized. Therefore, methods that can overcome this dilemma are urgently needed. This study proposes a method that applies a generative model to recognize masked faces based on face inpainting. We introduced a newly proposed identity loss term to conform to the identity information. The reconstructed face will be fed into a face recognition network to extract the feature embeddings for a distance comparison. Taking a naive generative model without an identity loss term introduced as the baseline, the model with an identity loss term improves the recognition accuracy by more than 4%. © 2022 IEEE.

REAL-TIME MASKED FACE REVEALING FOR VIDEO CONFERENCE

Video conferencing is an essential way for contactless conversation, which conveys abundant multimedia signals. Especially under COVID-19, the video conference has been becoming a common way for daily communications. However, for the sake of plague prevention, it usually happens that the people attending the video conference are wearing a mouth mask, leading to inconvenient communication due to incomplete facial information. To tackle this problem, we develop a novel system that reveals the masked faces in real-time, making each participant feel like the others are mask-free. Moreover, we map the audio to 3DMM expression to guide the generation of various mouth shapes utilizing multi-modal information. Extensive experiments validate the revealing effectiveness and better user experience of the system. Furthermore, by applying lightweight networks design, the proposed system can run in real-time. © 2021 IEEE